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INTRODUCTION 


In  early  FY88,  the  Defense  Technical  Information  Center  (DTIC)  initiated  a 
project  with  the  Logistics  Management  Institute  (LMI)  to  design  and  produce  a 
prototype  compact-disc  — read-only  memory  (CD-ROM).  The  project  team  evaluated 
commercial  CD-ROM  products  and  discussed  design  requirements  of  the  prototype 
disc  with  the  Resource  Sharing  Advisory  Group  (RSAG).  Based  on  specifications 
derived  from  this  analysis,  DTIC  established  a  contract  with  Online  Computer 
Systems  to  produce  a  prototype  disc.  Mastered  in  January  1989  and  containing 
approximately  200,000  citations  from  the  Technical  Reports  database,  the  disc  was 
evaluated  by  30  individuals  at  18  DROLS  sites. 

This  report  evaluates  the  prototype  disc  and  makes  recommendations 
concerning  a  production  disc.  In  Chapter  1,  we  summarize  responses  to  the 
evaluation  survey  sent  to  all  prototype  users.  In  Chapter  2,  we  discuss  problems 
which  the  project  team  and  test  sites  encountered  in  the  production  and  installation 
of  the  disc.  In  Chapter  3,  we  discuss  issues  which  DTIC  should  consider  before 
committing  itself  to  a  production  CD-ROM  product.  Complete  responses  to  the 
evaluation  survey  are  compiled  in  Appendix  A. 
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CHAPTER  1 


USER  REACTION 


a.  Survey  Administration 

DTIC  selected  the  original  CD-ROM  test  sites  —  AFWL,  NRL,  and  TRADOC  — 
from  among  interested  RSAG  members.  Additional  sites,  including  the  Naval 
Academy,  National  Security  Agency  and  Naval  Surface  Weapons  Center,  were 
selected  later.  All  test  sites  were  registered  DoD  DROLS  users,  had  already  installed 
a  CD-ROM  reader  at  their  site,  and  were  interested  in  testing  and  evaluating  the 
prototype  disc. 

We  mailed  questionnaires  to  the  test  sites  in  early  May,  except  the  TRADOC 
questionnaires  which  were  mailed  in  early  June.  Survey  forms  were  sent  to  28  sites. 

b.  Summary  of  Survey  Responses 

As  of  mid- August  1989, 30  individuals  from  17  sites  had  responded.  We  provide 
highlights  of  their  responses  below  and  a  compilation  of  ail  responses  in  Appendix  A. 

•  How  disc  was  used.  Twenty-four  of  the  respondents  used  the  disc  for 
bibliographies  searches,  16  for  document  identification,  3  for  cataloging,  13 
for  evaluation  purposes,  2  as  a  teaching  medium,  and  1  to  identify  the 
agency’s  holdings  in  DROLS. 

•  Other  CD-ROM  products  used/Comparison  with  DTIC’s  disc.  The  most 
widely  used  CD-ROM  product  among  the  test  sites  was  Bowker’s  "Books  in 
Print”  (10  sites).  The  only  other  product  used  by  more  than  1  site  was 
Ulrich’s. 

One  reason  we  selected  Online  Computer  Systems  to  master  the  DTIC  disc 
was  because  their  software  is  used  on  "Books  in  Print.”  Thus,  all  "Books  in 
Print”  users  would  immediately  be  familiar  with  the  user  interlace  on 
DTIC’s  disc.  For  this  reason,  when  asked  to  compare  the  DTIC  disc  to  other 
CD-ROM  products,  many  simply  said  that  it  was  "about  the  same”  or  "pretty 
similar.” 

•  Disc  users.  The  primary  users  at  the  test  sites  were  librarians,  particularly 
reference  librarians,  and  library  technicians. 
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•  Time  spent  using  disc.  Seventeen  respondents  used  the  disc  for  a  total  of  less 
than  5  hours,  four  used  it  for  5  to  10  hours,  and  eight  for  more  than  10  hours. 

•  Brand  of  CD-ROM  reader  used.  Of  the  20  sites,  13  used  Philips/LMSI 
readers,  5  used  Hitachi,  and  2  left  the  question  blank.  The  only  Sony  drive 
was  that  used  by  DTIC. 

•  Monitor/PC  configuration.  Eight  respondents  used  color  monitors  and 
19  used  monochrome.  All  sites  ran  the  CD-ROM  on  a  personal  computer 
having  a  hard  drive  and  at  least  one  floppy  disk  drive. 

•  Disc  installation.  Nine  respondents  had  trouble  installing  the  disc, 
primarily  because  they  lacked  the  necessary  device  driver.  (See  section  2b 
below  for  a  description  of  this  problem.)  Some  sites  also  needed  help  in 
modifying  system  files  on  their  PC.  (See  section  2a  below.)  Once  these 
initial  problems  were  resolved,  the  disc  ran  smoothly  at  all  sites. 

•  Documentation.  Except  for  disc  installation  procedures,  the  sites  were 
generally  satisfied  with  the  documentation. 

•  User  interface.  All  but  one  of  the  thirty  respondents  found  the  user  interface 
easy  to  learn.  Twenty-six  sites  also  found  it  easy  to  use. 

•  Most  useful  features  of  the  disc.  The  respondents  found  the  disc  useful  in 
several  ways.  Among  the  features  mentioned  were:  easy  to  use,  saves  time 
and  money  compared  to  DROLS,  expanded  search  features  (keyword/subject 
searching  of  abstract  and  other  fields,  ability  to  combine  search  strategies), 
ability  to  print  unattended  and  to  save  citations  on  disk,  browsing 
capabilities,  good  backup  to  DROLS,  additional  display  formats,  ability  to 
search  all  numeric  fields  simultaneously. 

•  Least  useful  features  of  the  disc.  Several  features  were  mentioned:  limited 
number  of  search  sets  which  can  be  retained  at  one  time,  lack  of  holding 
symbol  (field  49),  limited  timeframe  of  citations  cf  the  disc,  lack  of 
instruction  in  defining  custom  format,  inability  to  order  documents, 
inability  to  search  multiple  words  as  a  phrase,  lack  of  classified  data  and 
WUIS/IR&D  data,  and  inability  to  specify  ranges  for  Report  Date  searches. 

•  Functions  to  add  to  the  disc.  Relatively  few  suggestions  were  made,  despite 
the  many  limitations  identified  in  the  previous  question.  Among  those 
listed  were:  capability  to  order  documents,  expanded  time  periods,  a  "panic” 
key  to  stop  printing  of  citations,  mouse  interface,  hierarchy  function,  and  a 
standard  bibliographic  output  format. 

•  Price  of  disc.  The  majority  of  organizations  wanted  the  disc  produced 
quarterly  or  semi-annually  and  would  be  willing  to  pay  between  $300  and 
$750  for  an  annual  subscription.  A  few  sites  would  be  willing  to  pay  more 
than  $750;  none  would  pay  more  than  $1500.  A  disc  produced  only  once  a 
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year  would  be  much  less  useful  than  a  quarterly  or  semi-annual  product.  A 
few  sites  could  not  support  both  DROLS  and  CD-ROM. 

•  Single  or  multiple  disc  product.  Twenty  respondents  preferred  a  multi-disc 
CD-ROM  product  which  would  provide  from  10  to  35  years  of  DROLS  data. 
Seven  respondents  would  be  satisfied  with  a  single  disc  offering  the  most 
recent  5-6  years  of  data. 

•  CD-ROM  features  preferable  to  DROLS.  Respondents  named  several 
features  which  they  preferred  to  DROLS:  retrieval  speed,  no  time/cost 
pressure,  simpler  retrieval  formats  (e.g.  no  "@STR@”  format)  and  fewer 
commands  to  remember,  windowing  capability,  choice  of  novice  or  expert 
level,  ability  to  combine  search  sets,  truncation,  no  dov»utime,  posting 
browse  statements  to  the  search  workspace,  ability  to  change  display 
formats  easily,  use  of  mnemonics  in  place  of  role  codes,  and  ability  to  search 
abstracts. 

•  DROLS  features  preferable  to  CD-ROM.  Respondents  also  liked  many 
DROLS  features  better  than  the  CD-ROM:  access  to  WUIS,  ability  to  order 
documents,  more  current  data,  hierarchy  searching,  ability  to  display  in 
scroll  mode,  ability  to  search  multiple  words  without  overloading  search 
workspace,  and  less  clutter  than  window  interface. 

•  Comparison  of  CD-ROM  retrieval  times  with  DROLS.  The  majority  of  users 
said  that  the  CD-ROM  was  either  somewhat  or  significantly  faster  than  the 
DROLS  TR  database.  One  site  said  that  DROLS  was  faster  for  subject 
queries;  another  found  that  CD-ROM  was  significantly  faster  than  dial-up 
but  slower  than  dedicated  line. 

•  Other  comments  and  suggestions.  We  received  several  interesting  responses 
to  this  question.  One  site  suggested  that  the  password  typed  to  the  screen 
should  be  invisible;  another  suggested  that  a  training  manual  be  produced  to 
walk  a  user  through  a  search;  others  stated  that  CD-ROM  is  good  way  for 
giving  users  direct  access  to  DROLS  data  without  concern  for  online  costs. 
The  most  gratifying  comment  was  from  a  site  who  said:  "A  SUPER 
product  —  patrons  love  it.  If  we  had  a  complete  set  of  files  on  CD-ROM,  we 
would  not  need  dial-up  access,  which  we  have  now  only  as  a  back-up  to  the 
dedicated  system.”  Eight  respondents  did  not  reply  to  this  question. 


CHAPTER  2 


ISSUES  ENCOUNTERE  N  PRODUCING  AND  INSTALLING  DISC 


a.  Modifications  to  System  Files 

When  we  received  the  CD-ROM  disc  from  Online  Computer,  we  ran  the 
installation  program  and  attempted  to  access  the  disc.  A  cryptic  error  message 
appeared  on  the  screen.  After  several  more  unsuccessful  attempts  to  install  the  disc, 
we  called  the  vendor  and  learned  that  modification  to  two  system  files,  "config.sys” 
and  "autoexec.bat”,  was  necessary.  These  modifications  were  simple  to  make  but, 
without  extensive  computer  knowledge,  a  user  would  not  know  what  was  needed.  We 
had  assumed  that  the  installation  program  supplied  with  the  disc  would  be  sufficient 
to  get  the  disc  ready  for  use,  but  it  was  not.  Several  test  sites,  as  well  as  DTIC-E, 
encountered  this  problem. 

b.  Incompatible  Devices  Drivers 

Since  all  test  sites  had  already  installed  a  CD-ROM  reader  and  most  were  using 
commercial  CD-ROM  discs,  we  were  surprised  to  learn  that  some  sites  needed  a  new 
device  driver  to  run  the  DTIC  disc.  (A  device  driver  is  a  computer  program  that 
contains  the  necessary  low-level  instructions  to  access  data  on  a  disc.  It  is  installed  in 
memory  as  an  extension  of  MS-DOS.)  We  discovered  that  many  CD-ROM  discs  come 
with  their  own  device  drivers,  which  are  often  not  compatible  with  those  from  other 
publishers. 

We  learned  from  Online  which  drivers  were  needed  for  readers  installed  at  the 
test  sites. 

Since  all  TRADOC  sites  use  the  same  brand  of  CD-ROM  reader,  HQ  TRADOC 
was  able  to  provide  the  necessary  driver  to  all  sites.  Other  sites  with  driver  problems 
were  addressed  on  a  case-by-case  basis. 

c.  Vendor  Support  Included  with  Disc 

Upon  encountering  the  device  driver  problem,  we  hoped  to  obtain  the  necessary 
drivers  from  Online.  We  then  learned  that  drivers  are  generally  purchased  with 
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CD-ROM  hardware,  not  with  software  applications.  Since  Online  sells  only  Sony 
CD  hardware,  they  could  provide  only  Sony  drivers;  since  most  test  sites  had  either 
Philips  or  Hitachi  drivers,  Online  could  not  help. 

We  were  also  uncertain  how  much  installation  support  was  included  in  the 
contract.  We  paid  for  four  hours  of  support  to  install  the  reader  at  DTIC  but  nothing 
for  other  sites.  So  when  other  sites  had  difficulty  installing  the  disc,  we  relied  on 
Online  only  for  problems  which  we  could  not  solve  ourselves.  In  some  cases  they  were 
helpful,  in  other  cases  they  did  not  respond. 

d.  Test  Site  Personnel  Inexperienced  with  CD-ROM  and  Personal  Computers  (PCs) 

Many  test  site  personnel  who  attempted  to  install  the  disc  were  inexperienced 
not  only  with  CD-ROM  but  with  PCs  as  well.  When  we  told  sites  that  the  system  files 
must  be  modified,  some  said  that  they  did  not  know  how  to  use  (or  did  not  have 
available)  software  for  making  the  modifications  (including  word  processing 
software).  Many  personnel  were  also  unfamiliar  with  the  system  files  themselves 
and  needed  to  be  led  blindly  though  the  modifications,  not  really  understanding  what 
the  changes  meant. 

e.  Inaccurate  DROLS  Data 

On  the  disc,  a  user  can  view  inverted  indices  of  many  DROLS  data  fields.  Upon 
browsing  these  lists,  we  discovered  many  errors  in  the  TR  data.  Errors  ranged  from 
simple  spelling  mistakes  to  fields  whose  entire  contents  consisted  of  a  single  letter  or 
punctuation  mark.  A  decision  must  be  made  as  to  whether  DTIC  wishes  to  correct 
the  data  in  DROLS  or  to  correct  it  only  on  future  CD-ROM  discs. 

f.  Vendor  Delay  in  Delivering  Discs 

Online  Computer  originally  promised  to  have  the  prototype  discs  ready  in  mid- 
October  1988,  in  ample  time  for  the  Annual  DTIC  User  Conference  in 
November  1988.  Because  of  inadequate  computer  time  and  the  volume  of  TR  data  to 
be  mastered,  the  due  date  slipped  several  times.  The  discs  were  eventually  deli/ered 
in  January  1989.  Since  the  disc  being  produced  was  only  a  test  product,  the  delay  was 
tolerable.  However,  if  DTIC  produces  a  production  disc,  such  delays  might  be 
unacceptable. 


CHAPTER 3 


ISSUESTO  CONSIDER  CONCERNING  A  PRODUCTION  DISC 

a.  Production  Options 

DTIC  has  three  production  options  for  future  CD-ROM  discs:  (1)  establish  a 
contract  with  Online  Computer  Systems  for  all  services,  (2)  establish  a  contract  with 
another  systems  integrator  for  all  services,  or  (3)  perform  premastering  in-house, 
using  special  CD-ROM  hardware,  and  have  a  vendor  perform  only  mastering  and  disc 
replication.  Each  option  is  described  below. 

Option  #1 :  .Online  Computer  Systems 

DTIC’s  simplest  option  is  to  have  Online  Computer  produce  future  discs.  Online 
would  act  as  a  full-service  integrator,  as  they  did  for  the  prototype,  providing  data 
reformatting,  user  interface  customization,  premastering,  mastering,  and  disc 
replication.  If  requested,  they  would  also  provide  custom  documentation  and  limited 
training.  The  advantages  of  this  option  are  that  it  minimizes  risk  (since  Online  has 
demonstrated  their  ability  to  deliver  a  quality  product),  offers  relatively  known  costs 
(assuming  that  future  costs  do  not  differ  greatly  from  the  prototype  costs),  and 
provides  a  product  that  test  sites  found  useful  and  easy  to  use.  Since  Online’s  services 
may  no  longer  be  available  through  FEDLINK,  DTIC  may  need  to  establish  its  own 
contract. 

Option  #2:  Another  Systems  Integrator 

A  second  option  is  to  obtain  these  same  services  from  another  vendor.  This 
option  will  probably  be  more  expensive  than  the  first  because  the  new  vendor  would 
be  starting  from  scratch  and  would  need  to  develop  programs  to  reformat  DROLS 
data  and  customize  the  user  interface,  which  Online  has  already  done.  The  chosen 
vendor  must  offer  the  most  important  retrieval  features  offered  by  Online  as  well  as  a 
few  additional  features  named  by  the  survey  respondent:.,  such  as  an  expanded 
search  workspace. 
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Option  #3:  In-House  Development  and  Vendor  Mastering 


A  third  option  calls  for  DTIC  to  perform  data  clean-up  and  premastering  in- 
house  (using  a  product  such  as  "CD  Publisher”);  purchase  a  license  for  retrieval 
software;  and  have  a  vendor  master  and  replicate  the  discs.  Each  of  these  steps 
involves  special  requirements. 

First,  DTTC  would  purchase  "CD  Publisher,”  or  other  product,  to  prepare  a  data 
base  for  mastering  and  to  simulate  the  data  base’s  performance  on  CD-ROM.  "CD 
Publisher”  would  permit  DTIC  to  test  variables,  such  as  number  of  searchable  fields 
or  browsable  indexes,  to  achieve  maximum  performance  before  committing  to  a 
mastered  disc.  The  cost  of  a  1200  Mb  "CD  Publisher”  system  is  $41,815  from  the  GSA 
schedule. 

Next,  DTIC  would  license  a  retrieval  package,  such  as  KnowledgeSet,  Personal 
Library  Software,  or  HP  Laser  Retrieve,  to  use  with  the  disc.  (Since  Online 
Computer  does  not  license  their  software,  the  retrieval  package  used  with  the 
prototype  disc  would  not  be  available.)  Annual  license  fees  vary  greatly,  from  a  few 
thousand  dollars  up  to  $100,000;  some  companies  charge  a  fixed  annual  rate,  others 
charge  a  minimum  fee  plus  a  royalty  for  each  installed  site. 

DTIC  should  continue  relying  on  a  vendor  to  perform  the  final  step  —  mastering 
and  replication  —  because  of  the  very  expensive  equipment  and  highly  specialized 
personnel  required. 

Option  #3  would  require  funding  for  "CD  Publisher”  and  the  retrieval  software 
license,  as  well  as  for  additional  DTIC  staff  required  for  data  correction  and 
premastering,  data  clean-up,  and  user  support.  We  estimate  that  clean-up  and 
premastering  will  require  one  person-month  of  CD-ROM  staff  time  per  disc  —  thus,  a 
quarterly  product  would  require  four  person-months  per  year.  For  user  support,  a 
support  person  could  initially  divide  their  time  between  CD-ROM  and  DROLS.  As 
the  CD-ROM  product  becomes  more  widely  used  and  the  demand  for  support 
increases,  the  support  person  would  answer  only  CD-ROM  questions. 

DTIC  should  consider  option  #3  only  after  committing  itself  to  producing  an 
ongoing  CD-ROM  product,  developing  in-house  production  expertise,  and  exercising 
more  control  over  the  production  process.  Until  then,  DTIC  should  choose  one  of  the 
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first  two  options;  guidelines  for  choosing  a  vendor  are  provided  below  in  "d.  Choosing 
a  Vendor.” 

b.  Production  Costs 

The  cost  of  producing  the  prototype  was  approximately  $37,000.  This  included 
the  cost  of  database  development  (converting  more  than  6  years  of  data  from  DROLS 
format  to  Online’s  format,  indexing  the  data,  and  developing  format  displays),  user 
interface  modifications,  premastering,  mastering,  replication  of  100  discs,  8  software 
license  fees,  one  day  of  training  at  DTIC-SPO,  and  vendor  project  management. 

If  DTIC  were  to  add  CD-ROM  to  its  product  list,  two  types  of  costs  would  be 
incurred:  (1)  ongoing  and  (2)  one-time. 

The  primary  one-time  cost  would  be  for  customization  of  the  disc.  If  DTIC 
requires  extensive  changes  to  the  user  interface  of  the  retrieval  software  used  with 
the  disc,  the  changes  would  be  paid  for  once  but  would  be  available  for  all  future 
discs.  Since  only  minor  changes  were  needed  for  the  prototype  disc,  modification 
costs  amounted  to  only  a  few  thousand  dollars. 

Most  costs  of  a  production  product  will  be  ongoing;  that  is,  they  will  be  incurred 
would  the  production  of  each  newly  mastered  disc.  These  costs  include  database 
development,  premastering,  mastering,  replication  of  discs,  and  project  management. 
Two  of  the  prototype  tasks  listed  above  could  be  avoided.  First,  DTIC  could  spve  the 
cost  of  software  licenses  by  charging  users  for  it,  possibly  through  FEDLINK  as  was 
done  for  some  test  sites.  Second,  since  virtually  all  users  felt  that  the  disc  was  easy  to 
learn,  perhaps  DTIC  need  not  offer  formal  training,  thus  saving  the  training  cost. 
Subtracting  these  items  from  the  total  cost  of  the  prototype  leaves  a  balance  of 
between  $30,000  and  $35,000.  If  the  disc  were  produced  quarterly  by  Online,  the 
annual  cost  of  production  would  be  between  $120,000  and  $140,000.  If  another 
vendor  were  chosen,  the  costs  could  vary  greatly  from  these  figures.  If  DTIC 
purchased  "CD  Publisher”  and  paid  a  vendor  for  only  mastering  and  disc  replication, 
the  cost  for  those  tasks  would  be  between  $10,000  and  $15,000  per  year  for  a 
quarterly  product,  plus  the  cost  of  "CD  Publisher.” 
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c.  Pricing  the  Disc 


There  are  at  least  four  methods  for  DTIC  to  set  the  price  of  a  CD-ROM  product: 

•  Based  on  what  users  are  willing  to  pay.  DTIC  could  set  a  price  based  on 
what  it  believes  users  are  willing  to  pay.  Using  responses  to  the  evaluation 
survey,  users  appear  to  be  willing  to  pay  between  $300  and  $750  for  a 
subscription  to  a  disc  produced  quarterly  or  semi-annually. 

•  Based  on  DTIC  costs.  With  this  method,  DTIC  would  attempt  to  recoup  a 
predetermined  portion  of  its  costs  through  sales  of  the  disc.  At  first,  it  is 
unlikely  that  all  costs  could  be  recovered  because  the  disc  would  neither  sell 
enough  copies  nor  be  priced  estimating  the  number  of  customers. 

•  Based  on  the  average  price  of  commercial  CD-ROM  products.  DTIC  could 
also  price  its  product  in  line  with  other  bibliographic  CD-ROM  products. 
Given  the  price  distribution  of  currently  available  products,  DTIC  would 
price  its  disc  at  under  $1000  per  year  and  probably  closer  to  $500  per  year. 

•  Based  on  the  price  of  NTIS  CD-ROM.  The  CD-ROM  product  most 
comparable  to  DROLS  is  the  NTIS  disc.  Last  year  Dialog  began  offering 
NTIS  data  on  CD-ROM.  The  cost  is  $2,350  for  the  current  file  which 
includes  a  disc  containing  the  most  recent  four  years  of  data  (1985  —  89), 
quarterly  updates,  search  and  telecommunications  software,  documentation, 
telephone  search  assistance,  and  shipping.  The  complete  file  (data  from 
1980  to  present)  is  also  available,  for  $3,600. 

Silverplatter’s  version  of  NTIS  provides  the  last  five  years  of  data.  Its  cost  is 
$2,250  per  year,  and  includes  search  software,  documentation,  hotline 
support,  and  quarterly  updates. 

These  costs  are  much  higher  than  the  price  which  any  test  site  would  be 
willing  to  pay. 

d.  Selecting  a  Vendor 

We  used  the  following  guidelines  in  selecting  a  vendor  for  the  prototype  disc  and 
suggest  that  DTIC  do  likewise  for  the  production  disc.  We  believe  that  the  first  three 
items  are  especially  important  and  recommend  that  DTIC  place  special  emphasis  on 
them. 

•  Does  the  vendor’s  retrieval  software  offer  the  features  needed,  such  as 
keyword  access,  full  Boolean  searching,  truncation,  proximity  searches,  and 
numerical  range  searches? 

Given  DROLS  users’  extensive  experience  with  retrieval  software,  DTIC 
must  choose  a  full-function  package  which  offers  at  least  these  features. 
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Few  other  considerations  are  as  important  as  the  choice  of  retrieval 
software. 

•  Is  the  vendor’s  user  interface  easy  to  learn/does  it  make  intuitive  sense?  Is  it 
already  well  known  to  users  (through  other  products)?  Can  it  be  customized? 

The  simplicity  of  Online’s  user  interface  was  a  major  factor  in  our  choosing 
them.  We  found  "Books  in  Print”  (BIP)  very  simple  to  learn  and  we 
discovered  that  many  DROLS  users  also  used  BIP  and  were  therefore 
already  familiar  with  the  interface.  After  testing  the  prototype  disc,  every 
survey  respondent  said  that  it  was  easy  to  use. 

•  Does  the  vendor  offer  a  full  range  of  services  from  data  clean-up  and  user 
interface  modification  to  mastering  and  disc  replication? 

Until  DTTC  develops  in-house  capability,  one  vendor  should  be  chosen  to 
provide  all  needed  services.  That  vendor  may  need  to  subcontract,  as  Online 
did  for  disc  mastering,  but  DTIC  would  deal  only  with  the  prime  contractor. 

•  Has  the  vendor  mastered  other  bibliographic  databases?  Since  most  major 
integrators  have  mastered  bibliographic  databases,  this  requirement  should 
not  prove  a  limitation. 

•  Is  the  vendor  familiar  with  optical  disc  hardware?  Have  they  developed 
their  own  controller  cards?  Do  they  sell  optical  hardware?  Do  they  offer 
turnkey  systems,  including  installation,  training,  documentation,  and 
maintenance? 

We  believed  that  a  vendor’s  familiarity  with  all  aspects  of  optical  disc 
hardware  and  software  would  be  beneficial  to  the  project.  This  is  not  a 
crucial  factor,  but  a  useful  one. 

•  Is  the  vendor  familiar  with  the  DROLS  data  format  (or  with  the  COSATI 
format)? 

Online  Computer  had  worked  with  NTIS  data  and  was  therefore  familiar 
with  the  COSATI  format.  This  may  have  minimized  the  time  and  effort 
needed  to  develop  a  conversion  program  from  DROLS  format  to  Online’s 
format. 

•  Is  the  vendor  willing  to  examine  a  sample  of  DTIC’s  data  in  order  to  gave  an 
accurate  estimate  of  costs? 

Online  analyzed  a  sample  of  DROLS  data  to  estimate  the  overhead  rate  for 
inverted  indexes  and  to  determine  the  accuracy  of  the  data.  This 
information  enabled  them  to  estimate  production  costs  more  accurately. 

•  Are  the  vendor’s  services  available  through  a  central  procurement  source, 
such  as  FEDLINK,  or  will  DTIC  need  to  issue  an  RFP? 
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Using  the  FEDLINK  contract  simplified  the  procurement  and  allowed  us  to 
focus  on  the  technical  aspects  of  the  project.  This  service  will  probably  not  be 
available  through  FEDLINK  in  the  future. 

•  Is  the  vendor  located  in  the  Washington,  D.C.  area?  Since  Online  is  a  local 
company,  we  were  able  to  have  face-to-face  meetings  and  to  deliver/pickup 
material  on  short  notice.  Online  was  also  abie  co  offer  training  without 
charging  for  travel  costs. 

e.  Target  Users 

When  the  prototype  disc  was  demonstrated  at  the  DTTC  User  Conference  last 
October,  many  contractors  expressed  interest  in  obtaining  the  disc.  Since  the  test 
was  limited  to  DoD  DROLS  users  only,  we  were  not  permitted  to  provide  them  a  disc. 
We  recommend  that  DTIC  explore  the  possibility  of  making  a  CD-ROM  product 
available  to  all  registered  users,  including  contractors.  Doing  so  would  benefit  DTIC 
in  at  least  three  ways:  (1)  increased  use  of  the  disc  by  library  staffs  would  reduce  use 
of  DROLS;  this  could  result  in  improved  performance  for  all  online  users;  (2)  the  more 
copies  of  the  disc  that  are  sold,  the  more  likely  that  DTIC  can  recover  its  costs;  and  (3) 
based  on  the  positive  comments  received  about  the  prototype,  increased  distribution 
of  the  disc  would  enhance  DTlC’s  image  and  demonstrate  its  commitment  to  serving 
all  its  customers. 

f.  User  Support  Which  DTIC  Must  Offer 

DTIC  must  consider  offering  two  types  of  CD-ROM  support:  installation  support 
and  operations  support.  First,  based  on  the  experience  gained  from  the  prototype, 
users  will  have  many  problems  installing  the  disc  on  their  personal  computers.  Some 
of  these  problems  can  be  avoided  by  including  detailed  instructions  with  the  disc, 
especially  guidelines  on  modifying  system  files  (see  section  2b  above).  In  addition, 
before  a  new  CD-ROM  is  distributed  to  users,  DTIC  should  determine  which  device 
drivers  are  needed  to  run  the  disc  on  the  3  major  brands  of  CD  readers  (Hitachi,  Sony, 
Philips/LMSI)  and  should  provide  a  means  for  users  to  procure  the  necessary  drivers. 
(If  users  are  running  more  than  one  CD  application  on  a  single  personal  computer, 
they  may  need  to  load  a  different  driver  for  each  application.  DTIC  should  develop 
and  disseminate  a  procedure  for  doing  this.) 

Even  if  detailed  instructions  are  included,  many  inexperienced  users  will 
probably  have  problems  and  will  call  for  help.  DTIC  must  be  prepared  to  offer 
telephone  assistance  in  diagnosing  problems  and  suggesting  solutions.  Since  some 
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sites  do  not  hnv®  word  processing  software  on  the  PC  used  for  CD-ROM  and  do  not 
know  how  to  use  the  DOS  editor  (EDLIN),  DTIC  should  determine  a  method  of 
altering  system  files  which  all  users  can  utilize.  Since  most  IBM-compatible  personal 
computers  are  equipped  with  EDLIN,  we  recommend  that  DTIC  become  familiar 
enough  with  it  to  help  untrained  users  use  it  for  the  necessary  file  modifications. 

Second,  once  a  user  has  installed  the  disc,  DTIC  must  answer  questions  about 
its  use.  For  example,  users  may  have  questions  about  performing  searches,  browsing 
the  inverted  indices,  or  printing  a  bibliography.  These  questions  are  generally 
simpler  than  the  installation  questions  —  the  answers  would  often  be  found  in  the 
user  manual.  Of  the  questions  we  received  from  users  of  the  prototype  disc, 
installation  questions  outnumbered  operations  questions  by  a  ratio  of  at  least  10:1. 

g.  Suggested  Disc  Changes 

We  recommend  that  DTIC  personnel  closely  review  the  DROLS  data  to  be 
mastered  before  producing  the  next  CD-ROM  disc.  The  prototype  disc  contained 
many  data  errors  and  inconsistencies  because  DROLS  data  errors  were  not  detected 
and  corrected.  The  most  egregious  errors  on  the  prototype  disc  were  fields  whose 
value  consisted  entirely  of  punctuation  characters  or  which  included  an  initial  blank 
space.  Ideally,  these  errors  would  be  corrected  in  the  TR  data  base  before  the  data  is 
downloaded  for  mastering.  Alternatively,  the  errors  could  be  corrected  on  the 
vendor’s  computer  during  premastering.  The  simplest  way  of  detecting  these  errors 
is  to  review  the  inverted  indices  of  field  values.  Most  vendors  producing  customized 
CD-ROMs  encourage  the  customer  to  browse  the  inverted  indices  before  a  disc  is 
mastered. 

In  addition,  we  recommend  that  DTIC  evaluate  the  following  user-requested 
changes:  (1)  expand  search  workspace  to  permit  more  than  12  previous  searches,  (2) 
add  DROLS  search  hierarchy  to  the  disc,  and  (3)  devise  a  means  of  tagging  items  and 
transmitting  orders  to  DROLS  from  CD-ROM.  All  changes  would  need  to  be 
negotiated  with  Online  or  other  vendor  chosen. 

Other  recommended  changes  concerning  target  users,  user  support,  and  data  to 
master  are  discussed  in  other  sections  of  this  chapter. 
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h.  Data  to  Master 


For  its  next  disc,  we  recommend  that  DTIC  again  master  as  much  recent  data  as 
can  fit  on  a  single  disc  (approximately  5  to  6  years  of  TR  data).  Given  the  diversity  of 
interests  among  DROLS  users,  this  subset  of  the  TR  data  base  best  serves  most  users. 
Alternative  subsets,  such  as  citations  within  selected  field  and  groups,  would  not 
serve  as  many  users. 

Given  the  test  sites’  interest  in  historic  TR  data  (more  than  60  percent  of  the 
sites  expressed  interest  in  a  multiple  disc  product),  DTIC  should  consider  mastering 
at  least  2  discs  —  one  with  the  most  recent  5  — 6  years  and  a  second  disc  with  the 
previous  5  —  6  years.  5'our  respondents  requested  that  30  years  of  data  (or  the 
complete  file)  be  mastered;  this  volume  of  data  would  require  approximately  7  CD- 
ROM  discs. 

Several  users  also  requested  that  DTIC  produce  WUIS  and  IR&D  CD-ROMs  in 
addition  to  the  TR  CD-ROM. 

i.  Other  Recommendations 

Although  we  anticipate  that  DTIC  will  provide  direct  support  to  disc  users,  it  is 
unclear  what  support  the  vendor  will  offer  DTIC,  especially  for  problems  not  easily 
categorized  as  either  hardware-related  or  software-related.  Before  producing 
another  disc,  we  recommend  that  DTIC  establish  a  clear  understanding  with  the 
vendor  concerning  software  support. 

We  encountered  this  problem  with  the  prototype  disc  (concerning  incompatible 
device  drivers)  and  were  unsure  whether  Online  Computer  was  responsible  for 
assisting  us  or  not.  For  types  of  support  not  offered  by  the  vendor,  DTIC  may  need  to 
identify  other  sources  for  help. 

We  also  recommend  that  DTIC  install  three  CD-ROM  readers,  one  from  each 
major  manufacturer  (Sony,  Hitachi,  Philips/LMSI),  for  use  by  DTIC  CD-ROM  hotline 
support  personnel. 

Lastly,  we  recommend  that  DTIC  consider  including  penalties  in  the  vendor 
contract  for  delayed  delivery  of  discs.  Online  Computer  originally  promised  to 
deliver  the  prototype  discs  in  mid-October  1988  (in  time  for  the  DTIC  User 
Conference);  delivery  did  not  occur  until  early  January  1989.  A  shortage  in  Online’s 
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computer  processing  caused  the  delay.  Since  DTIC  was  mastering  over 
500  megabytes  of  data  (including  indices),  Online  needed  to  devote  its  only  processor 
to  the  disc  for  several  days.  Online  had  several  lower  volume  discs  to  produce  for 
other  customers  and  we  suspect  that  they  delayed  the  DTIC  disc  in  order  to  meet 
these  other  commitments.  To  avoid  similar  problems  in  the  future,  a  penalty  clause 
may  be  necessary.  This  could  not  be  done  with  the  prototype  disc  because  DTIC  used 
FEDLINK  contract,  which  could  not  be  modified. 

j.  Conclusion 

We  believe  that  a  DTIC  CD-ROM  product  would  be  useful,  well  received,  and 
image  enhancing.  We  recommend  that  DTIC  commit  itself  to  such  a  product. 

During  the  first  year,  DTIC  should  emphasize  user  support,  especially  for  first¬ 
time  CD-ROM  users.  As  discussed  earlier,  the  most  common  user  problems  will 
probably  involve  installation  of  the  disc.  DTIC  must  have  trained  user  support 
personnel  who  can  lead  naive  users  through  a  successful  installation  over  the 
telephone. 

Once  DTIC  has  established  sound  user  support,  it  should  analyze  alternative 
production  methods  and  decide  whether  to  develop  an  in-house  production  capability. 
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APPENDIX  A 

COMPILATION  OF  CD-ROM  E  VALUATUON  SURVEY  RESPONSES 


In  this  Appendix  we  provide  a  compilation  of  all  responses  to  the  CD-ROM 
evaluation  survey  distributed  by  DTIC  to  the  test  sites.  Each  respondent  is  assigned 
a  letter  code;  a  key  to  the  codes  is  provided  on  the  final  page  of  the  Appendix. 


/ 


1 .  How  is  the  CD-ROM  being  used  at  your  site? 


(ACDEFGHIJLMNPQRSTUVXYZ- 

a. 

For  bibliographic  searches 

AA,BB) 

(ACFGHIJLMPQRSTXY) 

b. 

For  document  identification 

(BCM) 

c. 

For  cataloging 

(GHIKOSUWXYZ,AA,CC) 

d. 

For  evaluation  purposes 

(FMT) 

e. 

Other  uses  (specify):  (F)  Inventory  of  our 
agency’s  documents  in  system;  (M) 
Demonstrate  to  other  information 
professional  and  to  combat  developers;  (T) 
Teaching , 

(DD)  [Blank] 


2a.  What  other  CD-ROM  products  do  you  use? 

Books  in  Print  (A)  (B)  (C)  (D)  (Q)  (R)  (S)  (T)  (U)  (W) 
Ulrich’s  (A)(B) 

Grolier’s  Encyclopedia  (A) 

Microsoft  Bookshelf  (A) 

ERIC  (E)  (F) 

OCLC  Search  CD450  -  ERIC  (O) 

NTIS  from  Dialog  (P) 

IEE/IEEE  on  Disc  (R) 

Wilson  Line  (R) 

Computer  Library  (R) 

PC-Sig(R) 

Reader’s  Guide  (U) 

Applied  Science  and  Technology  (U) 

NTIS  (U) 

Government  Printing  Office  (U) 

Bibliofile  (W) 

EBSCO  Serials  Directory  (W)  (X) 

LC-Subjects  (CDMARC  Subjects)  (W)  (X) 

None  (G)  (H)  (I)  (J)  (K)  (L)  (M)  (N)  (Y)  (Z)  (BB)  (CC)  (DD) 
Blank  (AA) 


2b.  How  does  the  DTIC  disc  compare  to  them? 

(A)  About  the  same,  have  same  format,  easy  to  use 

(B)  Same 

(C)  Pretty  similar 

(D)  Not  yet  installed 

(E)  Better  software  and  (apparent)  search  software 

(F)  Better  -  retrieval  is  faster;  more  online  immediate  help;  more  browsing  capability  - 
more  fields 

(G)  (H)  (I)  (J)  (K)  (L)  (M)  (N)  N/A 

(O)  Very  similar,  somewhat  more  complex,  but  then  the  records  are  more  complex 

(P)  We  find  the  Dialog  product  easier  to  use 

(Q)  Same  format 

(R)  Average  —  the  IEE/IEEE  would  seem  to  He  an  approach  DTIC  should  consider  seriously 

(S)  Similar  search  software 
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(T)  Pretty  similar 

(U)  Similar  to  Books  in  Print  in  appearance  (i.e.  same  pulldown  windows).  Provides  more 
information  and  searchable  fields  than  Reader’s  Guide  and  Applied  Science  & 
Technology 

(V)  Good 

(W)  Very  similar,  no  color  (dislike),  can  leave  without  going  to  action  menu  (like) 

(X)  Similar  -  which  is  very  good  as  carry-over  knowledge  can  be  used. 

(Y) (Z)(BB)  (CC)N/A 

(AA)  [Blank] 


3.  Who  used  the  DTIC  CD-ROM  at  your  site? 

(A)  Library  technicians 

(B)  Librarians  and  library  technicians 

(C)  Reference  Librarian/Document  Cataloger  (same  person) 

(D)  Administrative  technician 

(E)  Me  (librarian  provided  instruction  and  manual  and  I  used  software  and  CD) 

(F)  Library  staff  —  professional  and  technician  patrons 

(G)  (H)  (I)  Supervisory  Librarian  and  2  Reference  Librarians 

(J)  Library  staff  for  the  prototype/test  initially.  Plan  for  primary  use  of  CD-ROM  disc  by- 
library’s  patrons 

(K)  Myself 

(L)  Reference  Librarians 

(M)  Reference  Librarian 

(N)  Reference  Librarian  on  behalf  of  students  and  faculty. 

(O)  Librarian  only 

(P)  Librarians,  when  DTIC  report  file  is  down  (an  all-too-often  occurrence),  and  end  users. 
Library  offered  and  advertised  free  training  every  Tuesday  at  1400  for  anyone  who 
wanted  to  attend. 

(Q)  Reference  Librarian 

(R)  Library  personnel 

(S)  Three  reference/literature  searchers 

(T)  Reference  Librarians 

(U)  Library  Reference  Staff 

(V)  Reference  Librarians 

(W)  [Blank] 

(X)  Librarians  and  a  few  selected  patrons. 

(Y)  (Z)  (AA)  Reference  staff  only 
(BB)  Document  reference  personnel 
(CC)  [Blank] 

(DD)  N/A 


4.  How  much  time  did  you  spend  using  the  disc? 


(W,AA) 

a. 

Less  than  1  hour 

(CDHIJKLOQSTVX.CC, DD) 

b. 

Between  1  and  5  hours 

(RUZ.BB) 

c. 

5  to  10  hours 

(BEFGMNPY) 

d. 

More  than  10  hours 

(A)  Less  than  one  hour  a  day 
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5,  What  brand  and  model  of  CD-ROM  reader  did  you  install  the  DT1C  disc  on7  (circle  the 
appropriate  choice) 

Sonv  (None) 

Hitachi  (APRSTVWXYZ,BB,CC,DD) 

Philips/LMSl  (ACDFGHIJKLMNOQU) 

Blank  (E)  (AA) 


6.  What  type  of  monitor  did  you  Use  with  the  disc? 

(ABPQRSTU)  a.  color 

(CDEFGHIKLMORVWYZ-  b  monochrome 

BB,CC,DD) 

(AA)  IBlankl 


7.  What  type  of  drives  does  your  PC  have7 

All  sites  had  a  hard  drive  and  at  least  one  floppy  drive. 

No  sites  had  only  floppy  drives. 

8.  What  help  did  you  need  in  installing  the  disc?  (If  you  did  not  install  the  disc  yourself, 
answer  "N/A”.) 

(A)  Lots,  did  not  send  software  for  proper  installation 

(B)  Went  along  very  easily  with  materials  included 

(C)  (D)  (E)  N/A 

(F)  None,  once  it  worked  (modified  for  Phillips  CM100) 

(G)  Initial  installation  disk  did  not  provide  system  software  for  our  CD  ROM  player  Basic 
drive  installation  no  problem.  IM  office  installed  driver  After  proper  system  software 
was  received,  program  worked.  (We  are  a  TRADOC  library.) 

(H)  Initially,  package  from  DTIC  did  not  provide  system  software  for  our  CD  ROM  player 
(similar  problem  with  other  TRADOC  libraries  with  same  model  player)  Basic- 
installation  was  not  problem,  our  Academy  IM  office  personnel  installed  drive  After 
correct  software  was  received,  player/program  worked. 

(I)  The  initial  installation  disk  did  not  provide  function  of  the  system 

(J)  Assisted  by  the  School's  Computer  Training  Center  chief  after  additional  software  was 
received. 

(K)  1  didn’t  install  it,  but  I  had  to  help  reinstall  it  after  we  discovered  that  an  undefined 
custom  report  wouldn’t  retrieve  a  full  citation,  just  a  blank  window. 

(L)  Needed  assistance  with  changes  to  the  "config.sys”  and  "autoexec  bat”  files  Received 
help  from  Denm-*  O’Connor 

(MHNMO)  N/A 

(P)  No  help  from  DTIC,  so  I  had  to  figure  it  out  myself.  1  had  to  change  the  "config  sy>'  file 
All  in  all  not  an  unpleasant  or  difficult  experience. 

(Q)  N/A 

(R)  Driver  sent  by  DTIC  no  good  -  used  driver  sent  by  supplier  of  CD-ROM,  installed  with 
help  of  Computer  Library  CD-ROM  technical  personnel.  Online  service  poor 

(S)  Hitachi  purchased  from  Bowker  for  B1P  had  to  acquire  a  device  driver  for  DTIC  CD 
ROM  —  Bowker  driver  didn’t  work 

(T)  We  did  not  have  the  proper  drivers  to  load  the  disk  After  receiving  MS-DOS  Extensions 
we  were  fine.  This  caused  a  lot  of  confusion  and  time  'Mays 
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(U)  Several  phone  calls  to  DT1C  along  with  help  from  an  in-house  expert  on  CD-ROM  We 
needed  different  device  line  [in  system  fi.es]  in  order  to  operate  the  DTIC  CD-ROM  and 
still  operate  all  of  our  other  CD-ROMs. 

( V)  (W)  (X)  (Y)  (Z)  (AA)  (BB)  (DD)  None 
(CC)  [Blank] 


9.  Were  you  satisfied  with  the  documentation? 

(A)  Yes,  once  the  software  program  was  installed 

(B)  Yes 

(C)  Pretty  much 

(D)  [Blank] 

(E)  No.  Need  in-depth  support  to  take  advantage  of  "hidden”  features 

(F)  As  a  draft,  yes.  Forreal.no.  Needs  to  be  more  specific  in  some  areas.  Ex  how  to  go  back 
and  change  Custom  Format. 

(G)  On  the  whole,  yes.  Based  on  our  problems  with  installation,  need  to  give  complete  error 
listing  with  explanation  of  errors. 

(H)  Yes. 

(I)  Yes,  based  on  our  problems  with  installation.  There  is  a  need  for  a  complete  error  listing 

(J)  Yes 

(K)  See  answer  to  question  8  There  should  be  documentation  for  installation  that  explains 
what  the  result  of  a  choice  will  be. 

(L)  For  the  most  part,  although  I  felt  that  the  differences  between  the  search  and  browse 
modes  were  not  as  clearly  enumerated  as  they  could  have  been. 

(M)  Yes,  though  it  lacks  an  index,  and  table  of  contents  did  not  include  all  page  numbers. 

(N)  Problems  were  initially  encountered  interfacing  the  PC  and  the  CD-ROM  player.  We 
waited  several  weeks  before  the  device  driver  was  received.  However,  once  this  item  was 
received,  installation  was  easily  accomplished  and  the  documentation  was  satisfactory. 

(O)  Yes 

(P)  The  documentation  was  all  right  Could  give  better  examples  at  p.  inting  multiple 
citations. 

(Q)  Yes 

(R)  Installation  procedures  very  poor 

(S)  Yes 

(T)  The  problem  described  in  question  8  could  have  been  avoided  had  the  documentation  or 
DTIC  support  mentioned  the  need.  Error  in  documentation  concerning  function  key 
commands. 

( U)  Needed  more  help  than  what  was  provided  in  the  "Getting  Started”  section. 

(V)  Didn't  see  any  —  made  it  work  without  seeing  it. 

(W)  Did  not  use. 

(X)  Yes,  clear  and  concise. 

( Y)  Only  partially,  some  parts  seem  to  be  missing 

(Z)  Could  be  improved 

( AA)  Didn’t  know  we  had  documentation  when  l  searched  it. 

(BB)  Yes 
(CC)  [Blank] 

(DD)  N/A 

10.  Did  you  find  the  interface  easy  to  learn9 

All  sites  replied  "Yes”,  except  (Z)  which  offered  only  a  comment,  (AA)  who  answered  "No”  wi*h 
a  comment,  and  ( DD)  who  answered  "nfa.” 
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Comments: 


(E) 

II) 


(J) 

<M) 

<P) 

(R) 

(T) 

(U) 

(Y) 

(Z) 

(AA) 


I  am  a  computer  type  —  easy  to  leu  n/use 

Overall,  yes.  Tends  to  get  a  little  complex  fromone  window  to  the 
next. 

Probably  because  previous  searching  experience  was  with  DROLS 
direct,  online 

The  windows  he'ped  immensely  in  using  the  command  language 
Yes,  but  not  as  easy  as  the  Dialog  product. 

IEE/IEEE  and  Wilson  Disc  use  of  Basic  index  is  preferable  for 
average  user 

Had  exposure  through  BIP  and  online  searching  ( DIALOG) 

It  is  similar  to  Bowker’s  Books  in  Print  CD-ROM. 

System  has  advantages  and  disadvantages  [discussed  in  detail  on 
sheets  attached  to  this  survey. 

Too  time  consuming  to  use  compared  to  DROLS. 

I  didn’t  realize  I  had  gone  into  Expert.  I  should  have  told  it  to  go  into 
Novice. 


11.  Was  the  interface  easy  to  use? 


All  sites  replied  "Yes”,  except  (T)  who  answered  "yes”  and  "no”,  (Z)  who  responded  with  only  a 
comment,  ( AA)  who  responded  with  bewilderment  {''??”),  and  (DD)  who  responded  "N7A  ” 

Comments.  (E)  Need  mouse  interface.  Need  to  run  from  "Windows/Desqview” 

(F)  When  compared  to  dial-up  DTIC,  this  is  faster  and  easier  to  use  by 
observation,  since  I  have  never  used  dial-up  accession  to  the  system. 

(J)  Have  yet  to  evaluate  how  easy  s  find  it  to  use 

(T)  The  interface  is  easy  to  use  but  if  you  are  used  to  a  dedicated 

command  structure,  the  interface  is  a  bit  of  a  nuisance  Too  many 
windows,  too  slow,  too  many  keystroke  combinations  to  maneuver. 

(Z)  Awkward  to  use. 


12.  What  was  most  useful  about  the  disc7 

(A)  Easy  format  to  use,  search  time  options,  amount  of  time  saved  searching,  amount  of  ime 
online  saved 

( B)  Easy  formatting  of  printouts  and  ability  to  prini  unattended 

(C)  Mnemonic  format 

(D)  Great  money  savings.  Windows  extremely  nice.  Windows  make  it  easy  for  a  novice  like 
myself  to  search  expertly. 

(E)  Disc  is  not  useful,  software  is,  information  is.  What  was  most  useful?  access  to  DTIC 
citations.  I  could  torture  the  data  base  in  perverse  and  illogical  search  conjunctions  — 
different  combinations  of  keywords  found/unrovered  different  (and  useful)  information 
Great  feature:  saving  citations  to  floppy  for  word  processing. 

(F)  a.  Searching  by  keyword  and  report  number  in  multiple  fields  at  once 
b  Browsing  brief  citations-several  at  once 

c  Having  file  available  at  ALL  times  —  no  transmissions  problems,  DTIC  down  time, 
etc. 

d.  Mena  driven 

e  Windows  providing  search  commands  and  use  of  highlight  bars  to  select  rather  than 
rekey 

f.  Capability  to  print  and  to  download  to  disk 


(G)  Increased  speed  of  searching;  ease  of  searching;  ability  to  combine  search  strategies 
compared  to  online  searching. 

(H)  Searching  was  easier,  compared  with  dedicated  or  dial-up  modes.  Loved  all  those 
windows. 

(I)  Limitation  of  time  frame. 

(J)  Verification  of  document  limitations  and  classifications  prior  to  ordering;  subject 
searching 

(K)  Being  able  to  search  without  wondering  if  DT1C  would  be  up,  or  if  there  would  be  garbage 
on  the  phone  lines 

(L)  See  comments  to  question  18 

(M)  The  pull-down  windows  and  option  windows  accessible  at  almost  any  time  the  system 
was  running  was  the  most  useful  feature. 

(N)  Most  useful  was  disc’s  easy  accessibility  and  browsing  capabilities  without  concern  for 
online  costs,  telecommunications  problems,  etc.  Help  screen  were  very  useful  at  the 
novice  level.  Ability  to  activate  menus  with  "Alt”  keys  from  any  point  in  the  program 
was  also  useful. 

(O)  I  am  most  satisfied  with  the  CD  and  can  see  its  usefulness  to  "some  s.”  However,  it  is  no 
substitute  for  online  access.  If  utilization  can  justify  the  cost,  online  is  the  only  way  to  go. 

(P)  The  ability  to  have  a  backup  to  the  online  system  and  a  way  for  end  s  to  use  the  system 
without  using  our  terminal. 

(Q)  The  ability  to  have  access  to  DTIC  when  DROLS  connections  cannot  be  made. 

(R)  Ability  to  use  it  with  contractors  -  use  in  places  not  tied  to  modem  of  crypto 

(S)  Additional  displays,  defined  displays,  browse  mode,  searching  combined  author  and 
numeric  and  report  series  fields,  key  word  searches  each  field  key  word  indexed 

(T)  The  ability  to  set  up  a  strategy  before  going  into  dial  up  mode  (if  our  dedicated  line  was 
out).  Not  having  to  use  role  codes  (however  then  you  need  to  learn  the  indexes).  The 
numeric  field  designation  is  wonderful  as  is  abstract  searching. 

(U)  Using  browse  search  to  search  corporate  authors. 

( V)  Ease  of  use.  Help  screens  online  -  not  having  to  wade  through  a  manual. 

(W)  [Blank] 

(X)  N/A 

(Z)  Since  the  disc  contains  the  DROLS/DTIC  information,  why  was  the  DTIC  search  method 
not  used?9? 

(AA)  No  communication  problems;  the  files  available  whenever  we  needed  it. 

(CC)  I  liked  when  you  used  2  key  words  with  "and”  it  would  do  combination  steps 
automatically  for  you. 

(DD)  [Blank] 


13.  What  was  least  useful? 

(A)  Blank 

(B)  Subject  searching  limited  due  to  small  number  of  sets  which  can  be  retained  at  one  time. 

(C)  "9” 

(D)  Question:  Is  CD  as  current  as  DROLS? 

(E)  Need  to  be  able  to  customize  format  (no  info  in  manual).  Need  to  save  data  selected  after 
brief  citations. 

(F)  a.  Combining  sets  by  ”cs  =  ”;  this  was  cumbersome 

b  Limitation  of  12  sets  in  memory 

c.  No  holding  symbols,  field  49 

(G)  Limited  timeframe  of  documents  on  disc;  default  search  format  is  not  clearly  separated  on 
disc;  installation  procedures  would  indicate  that  user  would  not  see  "expert”  as  default 
setting  on  screens  after  "novice”  setting  had  been  chosen  upon  installation,  custom 
format  display  is  a  problem. 


(H)  Did  not  know  range  of  holdings  included  on  disk,  time  frame,  etc. 

(I)  Default  searching  format  is  not  clearly  separated  on  the  disk.  The  custom  format  feature 
presented  problem. 

(J)  For  staff,  the  use  of  "novice”  mode 

(K)  [Blank) 

(L)  Inability  to  order  documents  directly  (though  I  realize  this  is  impossible  without  an 
online  system). 

(M)  Cannot  search  multiple  word  phrases  as  keywords,  must  search  individual  words  and 
then  combine  them. 

Could  not  print  search  strategy  unless  used  the  computer’s  print  screen  function  -  and 
that  produced  a  messy  page. 

(N)  Least  useful  was  the  limited  screen  size  for  building  searches  —  disliked  having  to  delete 
Disliked  having  to  press  "F9”  in  the  novice  mode  before  starting  each  search. 

(O)  My  organization  requires  access  to  classified  data.  Can  this  be  accommodated? 
Quarterly  update  not  current  enough  for  our  purposes. 

(P)  The  limited  scope  of  the  product.  Would  like  to  have  all  DTIC  files  on  CD-ROM. 

(Q)  Title  search  was  awkward. 

(R)  Crosslink  to  inverted  file 

(S)  Incomplete  data  base,  WUIS  and  IR&D  data  bases  not  available,  DROLS  hierarchy 
feature  not  available 

(T)  Need  more  flexibility  in  the  Report  Date  query  -  to  select  ranges.  Too  many  windowing 
features  when  the  data  is  being  searched. 

(U)  Not  being  able  to  enter  a  title  without  having  to  break  it  up  into  keywords  to  search  for  it. 

(V)  (W)(Y)(AA)  [Blank) 

(X  N/A 

(BB)  Lack  of  space  for  search  terms 
(CC)  (DD)  [Blank] 


14.  What  functions  should  be  added  to  the  disc? 

(A)  Capability  to  order  documents,  bibliographies 

(B)  Expanded  time  periods 

(C)  Panic  "stop”  key  when  in  the  middle  of  printing.  "Esc”  didn't  seem  to  work.  May  I  just 
shut  off  the  printer  while  it’s  running? 

(D)  [Blank] 

(E)  Add  mouse  interface.  Add  Window/Desqview  interface.  Assign  unused  functions  keys  to 
"additional”  features. 

(F)  More  years  —  10  year  span  if  possible;  Field  49  —  holding  symbol;  See  19  below 

(G)  Sorting 

(H)  Sort 

(I)  None  right  off  hand.  Sorting  perhaps. 

(J) (K)  [Blank] 

(L)  Hierarchy  function 

(M)  Hierarchy,  if  possible. 

(N)  Uncertain  at  this  time, 

(O)  [Blank] 

(P)  N/A 

(Q)  [Blank] 

(R)  A  standard  bibliographic  citation  output  —  DTIC  online  output  is  the  worst  in  the  field 
and  was  preserved  on  the  disc  even  though  BIP  has  good  output. 

(S)  Hierarchy 

(T)  Not  necessarily  a  function  but  because  of  the  popularity  of  the  mouse,  all  IBM  windowing 
software  eventually  will  be  mouse  controlled.  There  are  so  many  different  software 
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packages  that  require  different  keystroking,  productivity  is  decreased  simply  by  having 
to  execute  the  commands.  Point  and  click  is  faster. 

(U)  Title  searching  in  the  browse  mode.  Title  searching  as  a  title  in  addition  to  the  keyword 
search  in  title  (i.e. ,  first  5  words  as  in  DROLS). 

(V)  (W)(X)(Y)(Z)  [Blank] 

(AA)  Combine  set  #’s  like  in  Dialog;  use  similar  search  techniques  as  Dialog;  Limiting  to 
individual  years  (i.e.  "L  +  /1987:1989”) 

(BB)  The  ability  to  search  more  terms  on  a  screen  and  display  results. 

(CCHDD)  [Blank] 


15.  Listed  below  are  3  possible  update  frequencies  for  a  DTIC  CD-ROM  product.  For  each 
frequency,  circle  the  price  that  your  organization  would  be  willing  to  pay  for  an  annual  subscription. 

a.  quarterly  (A)  $300-750  (FGHIJKN)  $750-1,500  (EPSTU)  $1,500 +  (AA) 

b.  semi-annually  $300-750  (DGHIJKQS)  $750- 1,500  (EU)  $1,500  +  (AA) 

c.  yearly  (C)  $300-750  (HJKMSU)  $750-1,500  (DEFGI)  $1,500+ (AA) 

H)  Can  there  be  a  reduced  price  for  DoD  School  facilities? 

(L)  Our  organization  cannot  guarantee  any  funds  for  future  purchases. 

(N)  Note;  it  is  doubtful  if  our  organization  could  pay  even  $7  50  per  year 

(O)  None  —  not  and  support  online,  too! 

(V)  (W)  (X)  (Y)  (Z)  (CC)  (DD)  [Blank] 


16.  Should  DTIC’s  CD-ROM  product  consist  of: 

(ADKLPU.BB) 

(BCEFGHIJMNPRST 
VXZ,AA,CC,DD) 

(OW) 

(B)  20  — 25  years 

(C)  25  years 

(E)  15  years 

(F)  30  years  -  complete  file 

(G)  (H)  (I)  25  years 

(J)  10  years 

(M)  All  years 

( N)  At  least  20  years 

(P)  All  years 

(R)  [Blank] 

(S)  10  years 

(T)  35  years 

(V)  20  years 

(Z)  All  years 

(AA)  30  vears 
(CCHDD)  [Blank! 


T 7.  On  the  prototype  disc,  retrieved  citations  are  displayed  in  reverse  AD  Number  order  Is 
this  acceptable7  If  not,  which  field  should  be  used  for  sorting  (in  place  of  AD  Number)? 


a.  a  single  disc  with  the  most  recent  5-6  years  of  data 

b.  multiple  discs  containing  recent  and  historic  data 
(If  you  selected  b.,  how  many  years  of  historic  data 
would  interest  you? 

[Blank! 
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All  sites  replied  that  AD  number  was  acceptable,  except  for: 

(C)  Report  date,  perhaps 

(F)  Report  date;  would  like  choice,  defined  by 

(G)  (H)  (I)  Optional  or  title,  author,  date 

(T)  Year/report  date 

(V)(W)(DD)  [Blank! 


18.  What  features  of  the  DTIC  CD-ROM  did  you  like  better  than  DROLS? 

(A)  Speed  of  information  retrieval 

(B)  Much  faster 

(C)  No  online  time  cost  pressure;  No  awkward  keying-in  @STR@  each  time;  Fewer 
commands,  etc.  to  memorize 

(D)  Windows.  Being  able  to  work  at  Novice  or  Expert  level.  Since  our  organization  deals 
with  several  databases,  it’s  sometimes  difficult  to  remember  all  operations  and  search 
keys.  The  Novice  format  simplifies  everything. 

(E)  [Blank] 

(F)  See  question  12  above;  also,  browsing  other  fields,  especially  Corporate  Author,  Report 
Number,  and  Keyword,  and  being  able  to  select  directly  without  having  to  retype 

(G)  Ability  to  combine  search  sets;  speed  of  system;  truncation;  ability  to  select  display 
records 

(H)  Ability  to  combine  search  sets;  windows,  improved  truncation 

(D  Same  as  survey  (G) 

(J)  Its  availability  to  Library’s  «•  ^ost-savings  when  used  by  staff  in  place  of  online  searches 

(K)  [Blank.  Respondent  never  searched  DROLS  —  see  question  21.) 

(L)  The  "CS”  mnemonic  which  allows  you  to  combine  the  results  of  previous  searches.  The 
lack  of  this  function  in  the  DROLS  database  was  my  biggest  complaint 

(M)  No  down-time,  speed  of  access.  In  some  cases  I  found  more  materials  on  the  CD-ROM 
than  a  similar  search  of  DROLS  (for  same  time-frame),  perhaps  because  of  broader  search 
using  individual  key  words. 

(N)  Ease  of  access  to  system  and  ability  to  browse  without  online  fee.  Posting  browse 
statements  to  search  workspace.  Collection  of  data  on  one  disc;  defined  custom  format; 
ability  to  change  and  choose  new  formats  quickly. 

(O)  Dependability 

(P)  Thank  you  for  not  entering  those  rotten  role  codes.  Being  able  to  search  all  numeric 
fields  at  one  time  is  wonderful.  Browsing  feature  is  great. 

(Q)  Quick  access 

(R)  Searching  —  standard  ASCII  file  downloaded 

(S)  See  question  12. 

(T)  No  role  codes,  ability  to  search  the  abstracts 

(U)  The  ability  to  search  corporate  authors  by  name,  not  source  code. 

(V)  Ease  of  retrieval.  No  complicated  hook-up, . .  .to  go  through. 

(W)  (X)(Y)  [Blank] 

(Z)  None. 

(AA)  Menus  are  helpful  in  CD-ROM;  thev  are  better  than  nothing. 

(BB)  None 

(CC)  (DD)  [Blank! 


19.  What  features  of  DROLS  do  you  like  better  than  the  CD-ROM? 

(A)  Ability  to  view  current  work  units,  order  documents 
IB)  None 
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(C)  More  current  (latest  information);  more  sophisticated  search  parameters 

(D)  Reingable  to  order  online 

(E)  [Blank] 

(F)  Hierarchy  search  —  corporate  author  and  DRIT  terms 

(G)  Continuous  scroll  display  of  records 

(H)  Ability  to  view  in  "scroll”  display  mode,  unlimited  holdings. 

(I)  Same  as  survey  (G). 

(J)  Searching  is  more  retrospective;  searching  techniques/protocols  seem  more  direct 

(K)  [Blank.  Respondent  never  searched  DROLS  —  see  question  21 .] 

(L)  Hierarchy  function 

(M)  Can  search  multiple  words  and  multiple  levels  without  overloading  save  space;  if  delete  a 
term  from  space  have  destroyed  search;  can  search  multiple  descriptors,  identifiers,  and 
keywords  in  one  search  with  fewer  keystrokes.  Can  search  hierarchy,  multiple  words 
and  levels,  entire  database  or  current  10  years,  can  print  search  strategy,  and  order  from 
search  result  immediately  or  cr^ fe  a  file. 

(N)  Greater  ability  to  build  search  -  disliked  limited  screen  workplace  on  CD-ROM.  Like 
ease  of  ordering  reports  online.  Access  to  full  database. 

(O)  Currency,  complete  files  available,  single  source  searching 

(P)  NONE.  .  except  of  course  that  it  has  all  the  recent  data  so  we  can’t  do  without  it. 

(Q)  Search  strategy 

(R)  Sorting;  depth  of  coverage 

(S)  See  question  13.  Also,  option  of  searching  either  recent  or  historical  data.  Switching 
files:  TR,WUIS,  IR&D. 

(T)  I  don’t  like  the  clutter  of  a  windowing  interface. 

(U)  The  ability  to  search  ALL  databases  in  DROLS  along  with  access  to  any  daily  reports. 

(V)  More  current. 

(W)  (X)  (Y)  [Blank] 

(Z)  All 

(AA)  None 
(BB)  Everything 
(CC)  (DD)  [Blank] 


20.  In  general,  how  did  CD-ROM  retrieval  times  compare  to  retrieval  times  using  DROLS? 


(F*JT*Z,BB) 

(LO*,AA) 

(CDF*NQSUV) 

(ABGHIMPRT*) 

(EKWX.CC.DD) 


a.  CD-ROM  was  slower  than  DROLS 

b.  CD-ROM  was  about  the  same  as  DROLS 

c.  CD-ROM  was  somewhat  faster  than  DROLS 

d.  CD-ROM  was  significantly  faster  than 
DROLS 

[Blank] 


*(F)  CD  was  slower  for  subject  queries  especially,  but  faster  for  some  fields 

*(0)  About  the  same  as  DROLS  when  DROLS  is  working,  which  these  days  is  most  of  the  time 

*(T)  CD-ROM  was  significantly  faster  than  dial-up  but  slower  than  dedicated  line. 


21.  Any  other  comments,  suggestions,  or  problems? 

(A)  (B)  (C)  (D)  Blank 

(E)  After  viewing  citations  (according  to  format)  you  return  to  "Brief  Citations”.  Selected 
Full  Citations”  disappear. 

Would  like  to  have  a  way  to  retain  selected  citations  upon  return  to  "search  workspace”. 
Use  Function  keys  to  allow  to  remove  selected  "brief  keys”. 


All 


[Note  from  Julie  Gibson,  Librarian,  concerning  survey  6:  Sorry  this  is  late,  but  one  of  our 
patrons  wanted  to  give  you  some  feedback.  Regardless  of  how  he  answered  the  questions, 
please  keep  in  mind  that  he  has  NOT  ever  used  the  dedicated  or  dial-up  himself  —  only 
through  the  library  staff. 

Regarding  questions  13:  We  never  got  the  Custom  Format  set  up  and  I  wasn’t  sure  how  to 
go  back  in  and  do  it  without  having  to  reinstall.  Manual  does  not  tell  how.  ] 

(F)  Get  a  CD-ROM  to  s  ASAP,  as  complete  as  possible  (all  years  of  TR  file  and  work  units) 
and  as  inexpensively  as  possible. 

A  SUPER  product  -  patrons  love  it. 

If  we  had  a  complete  set  of  files  on  CD-ROM,  we  would  not  need  dial-up  access,  which  we 
now  have  only  as  a  back-up  to  dedicated  system. 

Explore  possibility  of  classified  and  for  limited  distribution  CD-ROM  products  —  we  need 
classified  abstracts,  complete  descriptor  and  identifier  fields,  access  to  IR&D  file,  etc. 
Include  some  kind  of  symbol  to  show  that  info  has  been  omitted  (classified,  etc.)  so  knows 
more  could  be  obtained  from  dedicated/classified  system 

[The  following  note  was  attached  to  survey  6:  This  is  our  evaluation  of  the  CD-ROM  test. 
If  you  have  questions  regrading  my  comments,  please  call.  W'e  can’t  wait  for  the  "real 
thing”.  It’s  a  super  product.  Assuming  it  won’t  be  free,  we  hope  it  will  be  chargeable  to 
our  NTIS  account.  Please  include  U  S.  in  any  future  tests.  Signed,  Julie  Gibson, 
Librarian] 

[Julie  also  attached  a  note,  with  examples,  concerning  duplicate  AD  numbers  on  CD- 
ROM] 

(G)  A  training  manual  or  a  search  problem  that  would  walk  a  new  or  inexperienced  through 
a  complete  search:  the  problem,  the  search,  displays,  prints. 

(H)  Could  a  walk  through  search  be  included  for  new  users,  as:  problem,  search,  displays, 
prints. 

(I)  Same  as  survey  (G). 

(J)  [Blank] 

(K)  I  have  never  searched  DROLS,  only  been  around  people  who  have.  Also,  our  DTIC  disc 
was  only  installed  2  weeks  ago,  because  the  Supervisory  Librarian  and  I  are  both  very 
recent  transfers  to  this  library.  We  have  experienced  some  problems  with  the  CD  drive 
that  may  or  may  not  be  particular  to  the  DTIC  prototype. 

Also,  when  I  entered  "de  =  repair?”  I  got  7  hits;  when  I  entered  "de  =  repair”  I  got  1669 
hits.  There  seems  to  be  a  problem  with  using  the  "?”  for  truncation.  I  was  trying  to 
retrieve  "repair”  and  "repairs.” 

(L)  The  delay  in  getting  the  disc  operational  (see  question  8),  and  a  staffing  shortage  at  this 
library  meant  that  I  did  not  have  much  time  to  work  with  the  program.  However,  I  do 
believe  that  it  is  a  much  improved  system  over  DROLS. 

(M)  Password  typed  to  screen  should  be  invisible.  No  method  of  changing  password.  [This 
respondent  also  attached  a  copy  of  a  memo  (sent  to  HQ  TRADOC)  wmch  summarized  her 
responses  to  the  survey  ] 

(N)  [Blank] 

(O)  I  suspect  the  CD  product  could  be  of  use  to  some  s.  We  have  access  to  ERIC  online,  for 
example,  but  use  the  CD  version  almost  exclusively.  It’s  current  enough  for  most  of  our  s 
purposes  and  the  end  can  "play”  with  it  to  his  heart’s  desire  at  virtually  no  cost.  Neither 
of  these  advantages  apply  to  DTIC  on  CD  for  our  organization,  but  may  for  others. 

(P)  (Q)  [Blank] 

(R)  Classification  of  combined  subjects.  The  online  search  system  seems  to  be  slower  than 
most.  Limited  number  of  sets  (Wilson  Line  allows  over  35  sets  and  one  can  combine  7  in 
complex  Boolean  logic  at  once).  IEE/IEEE  logic  is  more  precise.  [NRL  also  attached  a 
letter  with  additional  comments  about  the  CD-ROM.) 

(S)  I  would  only  use  CD-ROM  if  DROLS  not  available,  to  search  number  fields,  authors,  key 
words  not  found  in  DROLS  and  for  all  key  word  searches.  Suggest  you  consider 
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producing  discs  for  COSATI  fields  and  groups  which  include  historical  data  back  to 
beginning  of  data  base. 

(T)  What  kind  of  CD-ROM  drives  will  you  support;  the  standalone  soft  5.25  inch  or  the  in 
chassis  hard  CD-ROM  drive.  If  l  didn’t  have  a  dedicated  line,  I  would  use  the  dial-up. 
However,  because  that  is  so  slow,  I  would  resort  to  the  CD-ROM  more  often. 

(U)  Additional  examples  of  searches  in  the  manual  and  a  keyword  index. 

(V) (W)  [Blank] 

(X)  Highly  recommended  for  staff  use  in  small  libraries  with  limited  funds  or  dial-up 
terminals. 

Highly  recommended  for  patron  use  in  libraries  with  staff  to  monitor  searches. 

(Y)  [Blank] 

(Z)  Not  enough  search  workspace;  search  options  are  too  cumbersome  to  use;  prefer  the 
DROLS  direct  approach  searching;  CD-ROM  could  be  beneficial  in  small  libraries  or  as  a 
backup  system;  DROLS  is  so  much  easier  and  faster  to  use. 

(AA)  (BB)  (CC)  [Blank] 

(DD)  Easy  access  for  searching  DTIC.  Very  comfortable  experience. 
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SURVEY  RESPONDENTS 


(A)  U.S.  Army  Military  Police  School  (Ramsey  Library),  Ft  McClellan,  AL 

(B)  Air  Force  Weapons  Laboratory,  Albuquerque,  NM 

(C)  TRADOC  Test  and  Experimentation  Command,  Ft  Ord,  CA 

(D)  U.S.  Army  Intelligence  School  Library,  Ft  Devens,  MA 

(E)  U.S.  Army  TRADOC  Analysis  Command,  White  Sands  MR,  NM 

(F)  U.S.  Army  TRADOC  Analysis  Command,  White  Sands  MR,  NM 

(G)  U.S.  Army  Sergeants  Major  Academy,  Ft  Bliss,  TX 

(H)  U.S.  Army  Sergeants  Major  Academy,  Ft  Bliss,  TX 

(I)  U.S.  Army  Sergeants  Major  Academy,  Ft  Bliss,  TX 

(J)  U.S.  Army  ICS  library,  Ft  Huachuca,  AZ 

(K)  U.S.  Army  OMMCS  Tech  Library,  Redstone  Arsenal,  AL 

(L)  U.S.  Army  SC&FG,  Ft  Gordon,  GA 

(M)  U.S.  Army  Ordnance  Center  and  School,  Aberdeen  Proving  Ground,  MD 

(N)  U.S.  Army  Logistics  Library,  Ft  Lee,  VA 

(O)  U.S.  Army  Armor  School  Library,  Ft  Knox,  KY 

(P)  Naval  Surface  Warfare  Center  (White  Oak),  Silver  Spring,  MD 

(Q)  U.S.  Army  IS  Technical  Library,  FT  Benning,  GA 

(R)  Naval  Research  Laboratory,  Washington,  DC 

(S)  Naval  Weapons  Center,  China  Lake,  CA 

(T)  Naval  Weapons  Center,  China  Lake,  CA 

(U)  National  Security  Agency,  FT  Meade,  MD 

(V)  U.S.  Army  Missile  Command,  Redstone  Arsenal,  AL 

(W)  U.S.  Army  Missile  Command,  Redstone  Arsenal,  AL 

(X)  U.S.  Army  Missile  Command,  Redstone  Arsenal,  AL 

(Y)  U.S.  Army  Missile  Command,  Redstone  Arsenal,  AL 

(Z)  U.S.  Army  Missile  Command,  Redstone  Arsenal,  AL 
(AA)  U.S.  Army  Missile  Command,  Redstone  Arsenal,  AL 
(BB)  U.S.  Army  Missile  Command,  Redstone  Arsenal,  AL 
(CC)  U.S.  Army  Missile  Command,  Redstone  Arsenal,  AL 
(DD)  U.S.  Army  Missile  Command,  Redstone  Arsenal,  AL 


; 
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